Skip to content

[Trellis] Weight Conversion that is compatible with Raiden - #5089

Open
YixuanWang-99 wants to merge 7 commits into
mainfrom
yixuann-debug-raiden
Open

[Trellis] Weight Conversion that is compatible with Raiden#5089
YixuanWang-99 wants to merge 7 commits into
mainfrom
yixuann-debug-raiden

Conversation

@YixuanWang-99

@YixuanWang-99 YixuanWang-99 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Description

This pull request introduces target-free weight conversion to WeightConverter and integrates it into MaxTextTrainingEngine for Raiden weight synchronization in Trellis / RL post-training workflows.
It enables the trainer to unroll scanned layers, execute inhomogeneous hybrid layer cycles, and prefuse/pad MoE weights directly on the trainer side—without requiring the rollout engine's target_state. Furthermore, it introduces significant host memory optimizations (including streaming piece-by-piece conversion and aggressive host cleanup via malloc_trim and gc.collect) to prevent host OOMs during device-to-host (D2H) staging, and makes previously silent failure modes in the Raiden path visible.

Key Changes

A. Target-Free Weight Conversion (weight_converter.py, convert_utils.py)

  • Key Synthesis (_build_target_free_plan): Derives target keys directly from source keys and config:
    • Homogeneous layers (cycle == 1): Maps layers $\to$ layers_{i}.
    • Hybrid cycles (cycle > 1): Maps (layers, layer_{slot}, ...) to layers_{b * cycle + slot} across blocks $b$.
    • MoE gate/up prefusion (fuse_moe): Pairs wi_0 and wi_1 and fuses them into wi (supporting both PER_SHARD_INTERLEAVE and CONCAT).
  • Target-Free Slicing & Fusion: Added _slice_bulk_target_free and _fuse_moe_bulk_target_free supporting both concrete JAX arrays and jax.ShapeDtypeStruct for abstract shape tracing.

B. MoE Kernel Alignment & Padding (moe_padding.py)

  • Added centralized helpers compute_padded_moe_mlp_dim() and next_power_of_two().
  • Automatically pads MoE intermediate sizes to satisfy the TPU GMM_v2 requirement.

C. Raiden Sync Integration & Memory Lifecycle (maxtext_engine.py)

  • Integrated Converter: prepare_weight_sync() invokes WeightConverter when use_weight_converter=True or vllm.use_weight_converter=True.
  • Metadata Caching: Caches staged metadata per train_step to prevent duplicate staging on repeated queries within the same step.
  • Aggressive Host Cleanup: Calls sync.release_host_arrays(), gc.collect(), and ctypes.CDLL("libc.so.6").malloc_trim(0) during staging and in release_weight_sync().
  • Pathways Support: Uses host_stage=is_pathways and safely manages host transfers to CPU for proxy backends.

D. Visibility of Silent Failures & Fail-Fast Diagnostics

  • Explicit Exceptions: Missing raiden_synchronizer or unrecognised staging_transport immediately raises descriptive errors with chained ImportError tracebacks.
  • Replicated Batch Dim Warning: Warns once if a batch dimension does not divide the mesh axis and falls back to replication.
  • Leaf Type Invariants: Ensures all transformed leaves are re-wrapped in nnx.Param so tensor keystr paths agree exactly with sampler-side nnx.State.

E. Sampler & Config Adjustments

  • Added rollout_backend ("maxtext" vs "vllm_torchax") to VLLM configuration in configs/types.py.
  • Gated Gemma-specific weight unrolling (unroll_gemma_scanned_weights) in MaxTextVllmSampler by model architecture.
  • Set stub_on_error_when_not_decoupled=True in gcloud_stub.py to prevent crashes when goodput stubs fail to import.

Tests

E2E Test passed. (2×2×2 v5p trainer, 2×2×1 v5p rollout, MAX_STEPS=2)
Set time around 9/3 3:00pm

Trainer logs

Rollout logs

Rollout responses are sensible like:

[RolloutNode] [collector] traj=traj_prompt_4_g0 completion_tokens=128 prompt_tokens=157 logprobs=128 text='\nWill catches 16 catfish and 10 eels, giving a total of 26 fish.\nHenry challenges him to catch 3 trout for every catfish Will catches. Since Will cau'

Metrics makes sense as well:

Step 0: loss: -0.0000 | reward_mean: 0.0625 | advantage_mean: -0.0000 | perplexity: 1.0000 | step_time: 53.64s
Step 1: loss: 0.0000 | reward_mean: 0.0000 | advantage_mean: 0.0000 | perplexity: 1.0000 | step_time: 15.46s

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for Qwen 3.5 hybrid cycle layers in the weight unscanning and synchronization pipeline, updates Raiden synchronizer import paths, refactors vLLM integration to use MaxTextVllmSampler, and introduces warnings for replicated batch dimensions. The code review feedback highlights a critical bug where self._raiden_syncs was accidentally removed from maxtext_engine.py's initialization, causing an AttributeError. Additionally, the reviewer pointed out outdated import paths in error messages and test probes, a risky string-stripping operation (rstrip('s')) in raiden_unscan.py, a potential unhandled case for cycle-slot matching, and an unused parameter in _fuse_and_unstack_moe.

Comment on lines 340 to +341
self._signature_compare_warned: bool = False
self._raiden_syncs: Any = None
self._replicated_batch_warned: bool = False

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

The initialization of self._raiden_syncs was accidentally removed from __init__ when adding self._replicated_batch_warned. This will cause an immediate AttributeError when prepare_weight_sync, release_weight_sync, or close is called. Please restore self._raiden_syncs: Any = None in __init__.

    self._signature_compare_warned: bool = False
    self._replicated_batch_warned: bool = False
    self._raiden_syncs: Any = None

Comment on lines +1176 to +1178
raise RuntimeError(
"staging_transport='raiden' requires tunix.experimental.worker."
"raiden_synchronizer, which the installed tunix does not provide. Install a"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The error message still refers to the old module path tunix.experimental.worker.raiden_synchronizer. Since the import path was updated to tunix.experimental.weight_sync.raiden_synchronizer, please update the error message to match the new path to avoid confusion during debugging.

Suggested change
raise RuntimeError(
"staging_transport='raiden' requires tunix.experimental.worker."
"raiden_synchronizer, which the installed tunix does not provide. Install a"
raise RuntimeError(
"staging_transport='raiden' requires tunix.experimental.weight_sync."
"raiden_synchronizer, which the installed tunix does not provide. Install a"

Comment on lines +41 to +43
try:
importlib.import_module("tunix.experimental.worker.raiden_synchronizer")
_RAIDEN_AVAILABLE = True

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test still probes the old module path tunix.experimental.worker.raiden_synchronizer to determine _RAIDEN_AVAILABLE. Since the engine now imports from tunix.experimental.weight_sync.raiden_synchronizer, this probe will evaluate to False even if the synchronizer is available at the new path, causing the test to bypass the real staging path. Please update the probe to use the new module path.

Suggested change
try:
importlib.import_module("tunix.experimental.worker.raiden_synchronizer")
_RAIDEN_AVAILABLE = True
try:
importlib.import_module("tunix.experimental.weight_sync.raiden_synchronizer")
_RAIDEN_AVAILABLE = True

# suffix and fold it into the global layer index below.
slot = None
if cycle_interval > 1 and suffix and isinstance(suffix[0], str):
match = re.fullmatch(rf"{re.escape(layer_container.rstrip('s'))}_(\d+)", suffix[0])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using rstrip('s') to strip the trailing 's' from layer_container is risky because it removes all trailing 's' characters (e.g., "class" would become "cla", "loss" would become "lo"). A safer and more robust approach is to only strip a single trailing 's' if it exists.

Suggested change
match = re.fullmatch(rf"{re.escape(layer_container.rstrip('s'))}_(\d+)", suffix[0])
match = re.fullmatch(rf"{re.escape(layer_container[:-1] if layer_container.endswith('s') else layer_container)}_(\d+)", suffix[0])

Comment on lines +127 to +128
expected_axis_len = num_reps if slot is not None else num_layers
if arr.shape[scan_axis] != expected_axis_len:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If cycle_interval > 1 but slot is None (e.g., due to a regex mismatch or a homogeneous parameter), expected_axis_len defaults to num_layers. However, the actual scanned axis length is num_reps. This mismatch will cause a confusing ValueError claiming a shape mismatch (expecting num_layers instead of num_reps), or an out-of-bounds error during slicing. Consider raising a descriptive error directly if slot is None when cycle_interval > 1.

Suggested change
expected_axis_len = num_reps if slot is not None else num_layers
if arr.shape[scan_axis] != expected_axis_len:
if cycle_interval > 1 and slot is None:
raise ValueError(
f"unscan_layers: {'.'.join(str(k) for k in key)!r} is missing the expected cycle-slot prefix "
f"under {layer_container!r}."
)
expected_axis_len = num_reps if slot is not None else num_layers
if arr.shape[scan_axis] != expected_axis_len:

Comment on lines 533 to 534
scan_fused_axis: int,
tgt_fused_axis: int,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The scan_fused_axis argument is no longer used in the new implementation of _fuse_and_unstack_moe. Since weight_converter.py is not part of this PR's diff, we cannot safely remove it from the signature without breaking the caller. However, please consider cleaning this up in a future refactoring of both files to remove the dead code and unused argument.

@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 10 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/maxtext/training_engine/maxtext_engine.py 35.71% 6 Missing and 3 partials ⚠️
src/maxtext/training_engine/checkpointing.py 0.00% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@YixuanWang-99 YixuanWang-99 changed the title [WIP] Debug Gibberish output after Raiden Sync [WIP] [Trellis] Weight Conversion that is compatible with Raiden Sep 2, 2026
@YixuanWang-99
YixuanWang-99 force-pushed the yixuann-debug-raiden branch 2 times, most recently from 0e43634 to 2e967ce Compare September 3, 2026 19:14
A9isha and others added 2 commits September 3, 2026 19:19
Each change targets a failure that produced no usable signal at the point of
cause: weights that never transfer, metadata that fails in another process, and
a performance cliff recorded only in a docstring.

unscan_layers had no tests, and it is the piece that decides whether trainer and
sampler tensor names agree. Both sides name tensors with jax.tree_util.keystr,
and nothing cross-checks the two sets: raiden_handler._validate_metadata only
validates one manifest's internal consistency (mesh rank, duplicate
variable/layer keys, sharding specs). A naming error therefore surfaces as
weights that silently never transfer. Two cases pin non-obvious invariants:
unscan_layers returns a plain nested dict while the sampler binds an nnx.State,
and keystr renders those identically only because the transform rewraps leaves
in nnx.Param -- dropping that rewrap would rename every tensor ("['k']" vs
"['k'].value"); and an already-unscanned state must raise, since without that
guard it would return unchanged and bind under scanned names.

prepare_weight_sync returned empty metadata on two paths: a missing
raiden_synchronizer (warning-level) and an unrecognised staging_transport (no
log at all). Neither is silent end to end -- WeightSyncCoordinator rejects an
empty side -- but the failure lands far from the cause, surfacing in another
process as "metadata collection returned an empty side", a count that never
names the missing module or the bad transport. The import case is the common one
rather than a corner: raiden_synchronizer ships only on tunix's Raiden branch,
so any released tunix takes it. Both now raise where the cause is known, with
the ImportError chained so the traceback keeps the module name. Because
staging_transport defaults to "raiden", this also reaches callers that never
asked for it, so the engine e2e test now probes for the synchronizer the way the
engine does -- exercising the real staging path where Raiden exists and the
documented failure where it does not.

_batch_data_shardings falls back to replicating the batch dimension when it does
not divide the batch axis's mesh size. That is correct -- every device along the
axis computes the whole micro-batch -- but it costs N times the work a sharded
one would do there. An invisible performance cliff is harder to notice than a
wrong number, because XLA's caching can make it look like nothing worse than a
slow run; the file already warns once per instance when a signature half cannot
be compared, and this extends that treatment. Warned once per instance rather
than per leaf, since the check runs under a tree_map over every loss input and
they normally share a batch dim. A sequence-packed micro-batch is always size 1
and has no alternative, so the message says the fallback may well be deliberate.

Verification. The unscan suite has teeth: renaming the emitted key from
layers_{i} to layer_{i} fails 5 of its 11 tests, including the name-equality
one. Marked post_training and left in tests/unit, which is already in
cpu-post-training-unit's path list, so the marker alone routes it -- tests/ and
tests/integration are not in that list, which is how the engine tests once ended
up collected by no job at all; collection confirms 11 tests in
cpu-post-training-unit and 0 in cpu-unit. The staging and sharding tests fail
without their respective changes. The sharding tests stub both the data spec and
the axis size: a single-device test mesh returns None in the batch position,
making the branch unreachable as configured, and an earlier draft asserted
`spec[0] is None` and passed without running the code under test at all.
- Support target-free key synthesis and unrolling in WeightConverter / MaxTextToMaxTextConverter for hybrid-cycle and MoE layers
- Add MoE padding utility for TPU GMM_v2 kernel alignment
- Cache staged weight sync metadata in MaxTextTrainingEngine and clean up host memory with gc and malloc_trim
- Add comprehensive TargetFreeConversionTest unit test suite
…rics recorder, and add cache invalidation

- Handle nested vllm dict/object in HyperParameters for use_weight_converter and rollout_backend
- Restore self._metrics_recorder = metrics_module.MetricsRecorder() in MaxTextTrainingEngine
- Invalidate staged metadata cache in release_weight_sync()
- Gate unroll_gemma_scanned_weights by Gemma model identity in MaxTextVllmSampler
- Set default num_lanes=128 in compute_padded_moe_mlp_dim
- Clarify memory lifecycle in WeightConverter convert docstrings and enhance test_case_5 memory profiling
…sync

- Add convert_streaming() to WeightConverter and MaxTextToMaxTextConverter for incremental transformation and eager memory release per group
- Add unscan_layers_streaming() to raiden_unscan with shared _unscan_one_key() helper
- Refactor MaxTextTrainingEngine.prepare_weight_sync() to stream piece-by-piece with unique strided worker indices
- Support RAIDEN_STREAM_PIECE_BATCH env var and deprecate RAIDEN_WEIGHT_SYNC_CHUNKS
- Add unit test coverage across weight converter, raiden unscan, and prepare weight sync suites
- Drop src_root ('base') prefix from target-free piece outputs in MaxTextToMaxTextConverter.convert_streaming()
- Remove out_root workaround in weight_converter_test.py test cases 1-4
- Remove dead code in WeightConverter.convert_streaming()
- Clean up _warned_raiden_sync_chunks check and simplify piece count mismatch validation in prepare_weight_sync()
…ase root in streaming converter

- Revert streaming piece-by-piece conversion in MaxTextTrainingEngine to single-piece convert and bind
- Restore base root prefix in MaxTextToMaxTextConverter.convert_streaming()
- Update prepare_weight_sync_test suite to reflect single sync instance
@YixuanWang-99 YixuanWang-99 changed the title [WIP] [Trellis] Weight Conversion that is compatible with Raiden [Trellis] Weight Conversion that is compatible with Raiden Sep 3, 2026
… and consolidate sync instances

- Under Pathways (JAX_PLATFORMS=proxy), require weight_synchronizer_ffi to avoid client host OOM
- Consolidate to single RaidenSynchronizer instance in MaxTextTrainingEngine
- Add reclaim_host_memory() utility invoking gc.collect() and malloc_trim(0)
- Add weight_sync_debug flag to HyperParameters config
- Update unit tests across maxtext_engine, prepare_weight_sync, and weight_converter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants